Evaluating the Feasibility of RAG Models for Regulatory Compliance in Investment Management

Project Summary

This project, led by Dr. Barry Quinn and Dr. Jesús Martínez del Rincón from Queen’s University Belfast, aims to create an AI framework to simplify and enhance regulatory compliance in global investment management.
First major project for Finance and AI Research lab (FAIR) established in 2023.
The research focuses on Retrieval-Augmented Generation (RAG) and ontology learning algorithms to transform complex regulatory texts into clear, consistent rules reflecting current international standards.
Study Period 1/11/2024 - 31/10/2024

Key Objectives

Explore the advantages of AI in regulatory compliance for the investment management sector.
Evaluate AI’s impact on accuracy, efficiency, and cost-effectiveness.
Focus on key areas like regulatory reporting, risk assessments, and compliance monitoring.
Address challenges in using Large Language Models (LLMs), including hallucinations, reasoning, and auditability.

Funding: UKRI through the UKFin+ program for a 12-month period; FEC £99,871 (Nov 2024 - Oct 2025).
Industrial Partner: Funds Axis Ltd provides support and industry insights.

Work Package 1: Use Case Definition, Data Collection & Process Mapping

Regulatory Focus

Codification of Investment Regulations:
- Focus on EFAMA European Fund Classification Categories.
- Rules for portfolio composition (e.g., asset class restrictions on equities, bonds, and country/industry limits).
- Compare regulatory limits with existing portfolios to ensure compliance.
Document Analysis:
- Use RAG to analyse multiple versions of regulatory texts for overlaps, inconsistencies, and conflicting rules.

Proposed Use Cases

Portfolio Compliance:
- Leverage Funds Axis’s classified spreadsheet for model training.
- Train models to validate portfolio adherence to regulations.
- Identify conflicting rules across versions.
Advanced Legal Analysis:
- Address complex regulations from authorities like the FCA.
- Extract logic from dense legal language for future use cases.
- For example Investment and Borrowing powers for UCITS from the FCA https://www.handbook.fca.org.uk/handbook/COLL/5/2.pdf

Client Engagement Opportunities

QA System Development:
- Address client portfolio queries using grounded references.
- Enhance credibility of responses through regulatory cross-referencing.

Next Steps

Define First Use Case:
- Review regulatory documents shared by Funds Axis.
- Develop the portfolio compliance framework for initial testing.
Plan In-Person Meeting:
- Visit Funds Axis to review workflows and processes.
- Map detailed use cases for inclusion in Work Package 1.

Timeline

Timeline of Project

Use Cases & Data Collection (Months 1-2)

Define precise regulatory and compliance scenarios.
Simulate investment firms using historical regulatory texts.
Compile datasets for training and evaluation.

Q/A Extraction (Months 2-3)

Develop a multi-hop question-answering system.
Fine-tune models like T5, Llama3.1, FinBERT.
Test for truthfulness and reasoning accuracy.

Rule Extraction with Ontology (Months 4-7)

Automate rule extraction using OWL ontology.
Integrate extracted rules into a dynamic knowledge base.
Ensure adaptability to evolving regulations.

Identifying Rule Set Inconsistencies (Months 8-10)

Use the MIMUS tool (McAveary et al. 2012) to detect inconsistencies in rule sets.
Resolve conflicts using Minimal Unsatisfiable Sets (MUSes).
Prioritise rules based on relevance, recency, and source.

Risk Assessment (Months 11-12)

Develop a compliance risk scoring system.
Leverage AI and data from company filings for risk prediction.
Produce a final compliance risk report.

Theoretical Framework in Economics

Artificial Intelligence (AI) as Adaptive Capital

A novel form of productive capital with unique properties that drive economic impact:

Key Economic Properties:

Dynamic Efficiency: Self-improvement through machine learning drives productivity.
Repurposability: Flexible application to diverse compliance tasks.
Scalability: Low marginal costs enable rapid scaling.
Labour Impact: Augments high-skill roles and automates repetitive processes.
Value Alignment: Aligns AI objectives with societal and economic goals.

Implications for Regulatory Compliance:

Reduced regulatory reporting times through dynamic efficiency.
Scalable solutions for cost-effective compliance across firm sizes.
Enhanced information access, reducing asymmetry in decision-making.

Economic Impacts of AI in Compliance

Efficiency Gains: Self-improving models reduce regulatory processing times.
Cost Savings: Scalability lowers compliance costs for SMEs.
Labour Market Impact: Augments high-skill roles, automates routine tasks.
Risk Mitigation: Flexible adaptation to regulatory updates reduces compliance risks.
Information Accessibility: Bridges knowledge gaps, enhancing decision quality.

Why This Matters:

AI transforms regulatory compliance into a scalable, efficient process.
Balances cost-effectiveness with ethical considerations like transparency and fairness.

Methodology Overview

Hybrid Economic Analysis Framework

Process Mapping: Establish baseline workflows and resource allocation.
Cost-Benefit Analysis: Quantify direct and indirect cost savings.
Simulation Modelling: Test adaptability of RAG models to evolving regulations.
Stakeholder Feedback: Gather practical insights on usability and challenges.

Ethical Considerations

Fairness: Minimise bias via curated datasets.
Accountability: Establish clear documentation for decision trails.
Human Oversight: Support compliance professionals without replacing roles.

Future Work

Train domain-specific language models for finance.
Enhance retrieval accuracy using semantic search.
Explore long-term cost and efficiency gains via real-world trials.

Thank You!

Contact Information

Dr Barry Quinn

Queen’s University Belfast

📧 b.quinn@qub.ac.uk